Rethinking Memory and Communication Cost for Efficient Large Language Model Training

Recently, various distributed strategies for large language model training have been proposed. However, these methods provided limited solutions for the trade-off between memory consumption and communication cost. In this paper, we rethink the impact of memory consumption and communication costs on the training speed of large language models, and propose a memory-communication balanced strategy set Partial Redundancy Optimizer (PaRO). PaRO provides comprehensive options which reduces the amount and frequency of inter-group communication with minor memory redundancy by fine-grained sharding strategy, thereby improving the training efficiency in various training scenarios. Additionally, we propose a Hierarchical Overlapping Ring (HO-Ring) communication topology to enhance communication efficiency between nodes or across switches in large language model training. Our experiments demonstrate that PaRO significantly improves training throughput by 1.19x-2.50x compared to the SOTA method and achieves a near-linear scalability. The HO-Ring algorithm improves communication efficiency by 36.5% compared to the traditional Ring algorithm

Ultrafast Radiographic Imaging and Tracking: An overview of instruments, methods, data, and applications

Ultrafast radiographic imaging and tracking (U-RadIT) use state-of-the-art ionizing particle and light sources to experimentally study sub-nanosecond dynamic processes in physics, chemistry, biology, geology, materials science and other fields. These processes, fundamental to nuclear fusion energy, advanced manufacturing, green transportation and others, often involve one mole or more atoms, and thus are challenging to compute by using the first principles of quantum physics or other forward models. One of the central problems in U-RadIT is to optimize information yield through, e.g. high-luminosity X-ray and particle sources, efficient imaging and tracking detectors, novel methods to collect data, and large-bandwidth online and offline data processing, regulated by the underlying physics, statistics, and computing power. We review and highlight recent progress in: a.) Detectors; b.) U-RadIT modalities; c.) Data and algorithms; and d.) Applications. Hardware-centric approaches to U-RadIT optimization are constrained by detector material properties, low signal-to-noise ratio, high cost and long development cycles of critical hardware components such as ASICs. Interpretation of experimental data, including comparisons with forward models, is frequently hindered by sparse measurements, model and measurement uncertainties, and noise. Alternatively, U-RadIT makes increasing use of data science and machine learning algorithms, including experimental implementations of compressed sensing. Machine learning and artificial intelligence approaches, refined by physics and materials information, may also contribute significantly to data interpretation, uncertainty quantification and U-RadIT optimization.Comment: 51 pages, 31 figures; Overview of ultrafast radiographic imaging and tracking as a part of ULITIMA 2023 conference, Mar. 13-16,2023, Menlo Park, CA, US

Spatially restricted drivers and transitional cell populations cooperate with the microenvironment in untreated and chemo-resistant pancreatic cancer

Pancreatic ductal adenocarcinoma is a lethal disease with limited treatment options and poor survival. We studied 83 spatial samples from 31 patients (11 treatment-naïve and 20 treated) using single-cell/nucleus RNA sequencing, bulk-proteogenomics, spatial transcriptomics and cellular imaging. Subpopulations of tumor cells exhibited signatures of proliferation, KRAS signaling, cell stress and epithelial-to-mesenchymal transition. Mapping mutations and copy number events distinguished tumor populations from normal and transitional cells, including acinar-to-ductal metaplasia and pancreatic intraepithelial neoplasia. Pathology-assisted deconvolution of spatial transcriptomic data identified tumor and transitional subpopulations with distinct histological features. We showed coordinated expression of TIGIT in exhausted and regulatory T cells and Nectin in tumor cells. Chemo-resistant samples contain a threefold enrichment of inflammatory cancer-associated fibroblasts that upregulate metallothioneins. Our study reveals a deeper understanding of the intricate substructure of pancreatic ductal adenocarcinoma tumors that could help improve therapy for patients with this disease

Comparing the efficacy of thyroglobulin and thyroglobulin/ thyroid-stimulating hormone ratio models in predicting a successful response to radioactive iodine therapy

Abstract Background The thyroglobulin (Tg)/ thyroid-stimulating hormone (TSH) ratio has manifested to be a reliable marker for predicting prognosis in patients with differentiated thyroid carcinoma (DTC). The objective of this study was to compare the efficacy of Tg and Tg/TSH ratio models in predicting a successful response to radioactive iodine therapy. Methods One thousand six hundred forty-two DTC patients receiving 131I radiotherapy were finally enrolled in this retrospective study. The patients were divided into a training set (n = 973) and a validation set (n = 669) by the patient consultation time (July 2019). A receiver-operating characteristic curve was constructed for Tg and the Tg/TSH ratio to establish their cutoffs. Then, the variables were screened by univariate logistic regression and incorporated into logistic prediction models by stepwise regression, where Tg/TSH was excluded from model 1 and Tg was excluded from model 2. Results In 1642 enrolled DTC patients, the first 131I radiotherapy had an excellent response in 855 patients. The cut-offs for Tg level and Tg/TSH ratio were 3.40 ng/ mL [area under the curve (AUC): 0.789] and 36.03 ng/mIU (AUC: 0.788), respectively. In addition, the AUC of the model including Tg was higher than that of the model including Tg/TSH in both the training set (0.837 vs 0.833) and the testing set (0.854 vs 0.836). Conclusions Both Tg and Tg/TSH ratios could be considered predictors of the effects of the first 131I ablative therapy. However, the prediction model including Tg performed better than the model including Tg/TSH

Targeted NGF siRNA delivery attenuates sympathetic nerve sprouting and deteriorates cardiac dysfunction in rats with myocardial infarction.

Nerve growth factor (NGF) is involved in nerve sprouting, hyper-innervation, angiogenesis, anti-apoptosis, and preservation of cardiac function after myocardial infarction (MI). Positively modulating NGF expression may represent a novel pharmacological strategy to improve post-infarction prognosis. In this study, lentivirus encoding NGF short interfering RNA (siRNA) was prepared, and MI was modeled in the rat using left anterior descending coronary artery ligation. Rats were randomly grouped to receive intramyocardial injection of lentiviral solution containing NGF-siRNA (n = 19, MI-SiNGF group), lentiviral solution containing empty vector (n = 18, MI-GFP group) or 0.9% NaCl solution (n = 18, MI-control group), or to receive thoracotomy and pericardiotomy (n = 17, sham-operated group). At 1, 2, 4, and 8 wk after transduction, rats in the MI-control group had higher levels of NGF mRNA and protein than those in the sham-operated group, rats in the MI-GFP group showed similar levels as the MI-control group, and rats in the MI-SiNGF group had lower levels compared to the MI-GFP group, indicating that MI model was successfully established and NGF siRNA effectively inhibited the expression of NGF. At 8 wk, echocardiographic and hemodynamic studies revealed a more severe cardiac dysfunction in the MI-siRNA group compared to the MI-GFP group. Moreover, rats in the MI-siRNA group had lower mRNA and protein expression levels of tyrosine hydroxylase (TH) and growth-associated protein 43-positive nerve fibers (GAP-43) at both the infarcted border and within the non-infarcted left ventricles (LV). NGF silencing also reduced the vascular endothelial growth factor (VEGF) expression and decreased the arteriolar and capillary densities at the infarcted border compared to the MI-GFP group. Histological analysis indicated a large infarcted size in the MI-SiNGF group. These findings suggested that endogenous NGF silencing attenuated sympathetic nerve sprouting and angiogenesis, enlarged the infarct size, aggravated cardiac dysfunction, and potentially contributed to an unfavorable prognosis after MI

AZD3759, a BBB-penetrating EGFR inhibitor for the treatment of EGFR mutant NSCLC with CNS metastases

Non-small-cell lung cancer patients with activating mutations in epidermal growth factor receptor (EGFR) respond to EGFR tyrosine kinase inhibitor (TKI) treatment. Nevertheless, patients often develop central nervous system (CNS) metastases during treatment, even when their extracranial tumors are still under control. In the absence of effective options, much higher doses of EGFR TKIs have been attempted clinically, with the goal of achieving high enough drug concentrations within the CNS. Although limited tumor responses have been observed with this approach, the toxicities outside the CNS have been too high to tolerate. We report the discovery and early clinical development of AZD3759, a selective EGFR inhibitor that can fully penetrate the blood-brain barrier (BBB), with equal free concentrations in the blood, cerebrospinal fluid, and brain tissue. Treatment with AZD3759 causes tumor regression in subcutaneous xenograft, leptomeningeal metastasis (LM), and brain metastasis (BM) lung cancer models and prevents the development of BM in nude mice. An early clinical study in patients with BM and LM treated with AZD3759 confirms its BBB-penetrant properties and antitumor activities. Our data demonstrate the potential of AZD3759 for the treatment of BM and LM and support its further clinical evaluation in larger trials.

Transduction efficiency after intramyocardial injection of NGF siRNA <i>in vivo</i>.

<p>(A) Representative site of intramyocardial injection (arrowed) and GFP expression in infarcted hearts of rats in the MI-SiNGF group at 1 wk time point. (B) Relative expressions of NGF mRNA, detected by real-time quantitative RT-PCR, in the sham-operated, MI-control, MI-GFP and MI-SiNGF groups at various time points (1, 2, 4 and 8 wk) after intramyocardial injection of NGF siRNA. Relative gene expressions of NGF were analyzed by the 2^-ΔΔCT method taking those in the sham-operated groups as 1. (C) Expressions of NGF (27 KDa) and GAPDH (36 KDa), analyzed by Western blot, in the sham-operated, MI-control, MI-GFP and MI-SiNGF groups at various time points (1, 2, 4 and 8 wk) after intramyocardial injection of NGF siRNA. (D) Relative protein expressions of NGF. The protein expression levels in (C) were quantified with Quantity AlphaEaseFCTM imaging software. Relative expression of NGF was normalized to GAPDH. Data were presented as mean ± SD. *<i>p<0.05</i> MI-control group vs. sham-operated groups, †<i>p<0.05</i> MI-SiNGF group vs. MI-control or MI-GFP groups. The results showed that expression levels of NGF mRNA and protein were induced in the MI-control group compared to the sham-operated group. NGF mRNA and protein in the MI-GFP group had no significant levels compared to the MI-control group, while those in the MI-SiNGF group were reduced compared to the MI-GFP group.</p